A Simulation Study to Evaluate the Robustness of Recent Methods for Preparing Variance Estimates in the Presence of Hot Deck Imputation

Many large-scale surveys currently use a variety of single imputation methods–as discussed by Chapman (1976), Cox (1980; and Kalton and Kasprzyk (1986)—to handle item nonresponse. Since the use of such imputation increases the underlying variation in the survey results, methods are needed to assess the impact. Until fairly recently, methods to assess the impact of the imputation on the variance have not been available. Rao and Shao (1992) and Shao (2002), presented a method to measure the variance of an estimate due to the combined effect of the sample design and the use of imputation to compensate for item nonresponse. This method discussed in section II, is based on the use of a replication method of variance estimation combined with specific adjustments to the imputed values. Our research sought to explore the use of this method on the expenditure data collected in the Medical Expenditure Panel Survey (MEPS), sponsored by the Agency for Healthcare Research and Quality (AHRQ). Since the MEPS utilizes a single hot deckimputation method, and the sample design and the data files were structured to facilitate the use of a replicate variance estimation method, the survey met the basic requirements to apply Shao’s method. This paper presents results from a simulation study conducted to evaluate the feasibility of applying Shao’s procedure to MEPS data. We will discuss, that the MEPS imputation procedures do not meet all of the methodological assumptions given by Shao. In particular, Shao’s method assumes the covariates used in the imputation process are fully reported and that variance estimates are needed only for univariate statistics. The goal of our simulations was to quantify the biases in estimates of variance when these assumptions were violated.